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Abstract 

Post-secondary educational institutions use various means to evaluate the teaching performance 
of faculty members. There are benefits to effective faculty evaluation, including advancing the 
scholarship of teaching and learning, as well as improving the functionality and innovation of 
courses, curriculum, departments, and ultimately the broader community (Boyer, 1990; Glassick, 
Huber, & Maeroff, 1997). While there is a body of research related to the evaluation of faculty in 
traditional settings, there have been fewer studies examining online faculty members’ 
perceptions of evaluation processes. Further, due to the growth of online education, the existing 
evaluation scales, including those used in traditional settings, have been questioned (Berk, 2013; 
Hathom & Hathorn, 2010; Rothman, Romeo, Brennan, & Mitchell, 2011). This qualitative study 
examines one university’s online full-time faculty and their perceptions of the tools and 
processes used to evaluate their teaching. Through a systematic content analysis of survey data, 
findings indicate that online faculty members have a desire to grow as instructors, infrequently 
focusing on modality or job expectations as a means for growth. Participants expressed an 
interest in holistic, descriptive evaluation feedback by a range of stakeholders, particularly those 
with content knowledge. Study findings have implications for administrators and other 
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stakeholders related to online full-time faculty, including the processes and documents through 
which they are evaluated. 

Keywords: evaluation, online faculty, full-time faculty, faculty evaluation, online learning, 
e-leaming, computer mediated learning 

INTRODUCTION 

The number of students participating in online learning is striking: 6.7 million students 
are taking at least one online course and 86.6% of colleges and universities now offer online 
courses (Allen & Seaman, 2013). Universities and colleges are turning to distance learning to 
meet the needs of students who seek a flexible, adaptable learning environment (Ragan, 2009). 
The surge in distance learning intensifies the need for institutions of higher education to develop 
effective evaluation processes for online faculty, as faculty evaluations can have an impact on 
teacher effectiveness, success, and growth (Berk, 2013; MacMillan, Mitchell, & Manarin, 2010; 
Wellein, Ragucci, & Lapointe, 2009). If faculty are evaluated effectively, there are a variety of 
benefits, including advancing the scholarship of teaching and learning, as well as improving the 
functionality and innovation of courses, curriculum, departments, and ultimately the broader 
community (Boyer, 1990; Glassick, Huber, & Maeroff, 1997). 

For effective evaluation to occur, evaluators must draw on multiple sources of data, 
rather than a single source, leading faculty to collaboration, reflection, and inquiry (Boyer, 1990; 
Glassick et al., 1997). Robust, well-constructed evaluations that encourage inquiry into one’s 
teaching practices can influence faculty to become rounded, productive members of the 
professoriate (Boyer, 1990; Glassick et al., 1997; MacMillan et al., 2010; Wellein et al., 2009). 
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Despite a broad acceptance that effective evaluation tools should be developed for and with 
faculty, to date, faculty evaluation systems have been largely insufficient (Arreola, 1979, 1986, 
1995, 2000a, 2000b; Arreola, Aleamoni & Theall, 2001; Berk, 2013). This is particularly true in 
the online environment where evaluation tools are often drawn from traditional settings, despite 
the arguably unique skills required to teach online (Berk, 2013; Hathom & Hathom, 2010; 
Rothman, Romeo, Brennan, & Mitchell, 2011). Research has asserted that online and traditional 
teaching techniques may be similar, but that there are key differences in evaluating online 
teacher effectiveness (Berk, 2013; Harrington & Reasons, 2005; Loveland, 2007). For instance, 
Hathom and Hathorn (2010) stated that the Web gives online instructors the opportunity to 
modify documents for students by including external links to additional resources. This is a 
unique skill to online teaching that can be evaluated by administrators. Other research has argued 
that there is little difference between teaching online and offline and therefore these evaluations 
should resemble one another (Berge & Meyers, 2000; Clark, 1989). As online education 
continues to grow and move through its generations (Moore & Kearsley, 2012), a deep and 
comprehensive understanding of online teaching and online faculty evaluation and its 
complexities is necessary. 

This paper examines a qualitative study of one university’s online full-time faculty and 
their perceptions of the tools used to evaluate their teaching. The authors provide a theoretical 
framework based in Lave and Wenger (1991) and Wenger’s (1998, 2000) notions of community 
of practice. The literature review explores faculty evaluation processes, specifically online 
faculty evaluation processes, as well as faculty commitment to evaluation processes. The study’s 


context, participants, and methodologies follow. The paper concludes with a report of the 
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findings and an analysis of the data’s broader implications for online full-time faculty and 
administrators in online departments. 

As more universities expand into online learning, it is critical to examine how faculty 
members are evaluated. Online full-time faculty’s perceptions of evaluation offer a window into 
the practices associated with evaluating faculty in this expanding learning modality. Further, it is 
necessary to gain insight into the functionality, effectiveness, and efficiency of evaluation tools 
for online full-time faculty. 

THEORETICAL FRAMEWORK 
Communities of Practice 

Examining the evaluation processes of online full-time faculty necessitates an 
understanding of this study’s theoretical framework. Lave and Wenger (1991) and Wenger 
(1998, 2000) contend that faculty members can form a community of practice. Three components 
combine to create a community of practice, including domain, community, and practice. The 
domain is represented by a shared interest. Members of the community engage in activities and 
discussions together to pursue their domain of interest. These members of the community then 
partake in practice together, ultimately leading to shared resources and stories. Communities of 
practice form in a variety of modalities, settings, and loci. Faculty members form a community 
of practice through alignment with a social and cognitive group, as well as lived experiences. 

Learning is a key component of a community of practice. Historically, it was assumed 
that learning occurred within an individual; however, further investigation has demonstrated that 
learning is social and exists entwined with experience, activity, and community (Lave & 

Wenger, 1991; Smith, 2003). Learning consists of complex social, cultural, and historical 
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systems that are accumulated and shared over time through participation within social learning 
systems (Wenger, 2000). The survival and success of a community or organization can be 
directly related to the knowing and learning that occurs within these social systems (Wenger, 
n.d.). 

In addition to the need for organizations to thrive, there is also a desire for increased 
innovation within organizations. Because of this, groups look to improve performance. To ensure 
learning is innovative, it must occur within and amongst members of a community of practice 
(Wenger, n.d.). Furthermore, individuals in a community must engage with one another to define 
competence and to improve meaningful knowing through various elements, such as joint 
enterprise, mutuality, and shared interests (Wenger, 2000). Communities of practice have been 
described as a process where active learning takes place due to the socialization of members that 
share in similar activities, ideas, and practices (Lave & Wenger, 1991; Smith, 2000). In turn, 
learning takes place within a community when relationships are experienced with other 
members, resulting in ingenuity (Wenger, n.d.). 

Competence and experience can work together to generate learning and innovation 
(Wenger, 2000). In addition, organizations can encourage cross-disciplinary projects to combine 
competence and knowledge from a variety of practices to assist in the production of a project or 
goal (Smith, 2003). The simultaneous learning that occurs from communities of practice can 
produce a loop that enhances learning and innovation, continuing well past the original project 
(Smith, 2003). In order for simultaneous social learning to be successful, a community of 
practice must include development that matters to individuals through a “shared repertoire of 
ideas, commitments, and memories” (Smith, 2009, para 13). Through social learning and other 
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similar interactions, individuals are bound within a relationship that cultivates trust. Because of 
this, groups are able to undertake greater and more complex activities over time (Smith, 2003). 
These processes can provide advantages to organizations like universities, as well as its faculty, 
as this brings forth greater effectiveness and success. 

Organizations like universities benefit from the social learning that can emerge from 
communities of practice (Smith, 2003). Not only do communities of practice assist in learning, 
but they also enable individuals within a group to take collective responsibility for managing 
knowledge needed to succeed. Through this understanding, organizations have found a direct 
link between learning and performance (Wenger, n.d.). When individuals learn, and do so 
collectively, performance improves. Because of this, many organizational development circles 
have shown a growing interest in utilizing community practices to overcome potential problems 
or challenges, as well as to encourage innovation and drive the application of learned knowledge 
(Smith, 2003). 

The goal of every university is to improve faculty to improve student learning. 
Universities can develop communities of faculty that learn from one another, engage in 
productive discussions, and collectively establish meaningful evaluation processes. This requires 
the development of processes that enhance faculty learning and investment in its programs, 
regardless of whether they teaching online or in traditional environments. Online faculty 
members are not individuals teaching in isolated environments, particularly in the environment 
discussed in this study; rather, they are a working organism, a community, a network, and a 
group that works collaboratively to ensure successful outcomes. If learning is contingent on 


community and community relies on cognitive and social experiences, then studies must 
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examine how online faculty learn to be better online faculty. Universities can use online faculty 
feedback and knowledge to develop evaluation tools that address the issues present in the 
environment. If evaluations are intended to improve online faculty performance, then it is 
incumbent upon universities to understand how online faculty members perceive the evaluation 
process and its associated documents. 

LITERATURE REVIEW 
Evaluations of Faculty 

Educational institutions use various means to evaluate the teaching performance of 
faculty members. Effectively constructed teaching evaluations allow faculty to identify strengths, 
as well as areas of opportunity in instructional practices. According to Wellein et al. (2009), an 
effective evaluation of faculty includes systematic assessment and reflective critique by various 
stakeholders, including peer, self, and specialists. MacMillan et al. (2010) contended that 
extensive evaluation mechanisms not only improve day-to-day teaching practices for individual 
instructors, they are also the first step to informed teaching and scholarship. Further, involvement 
of faculty in each step of the creation, development, and implementation has been found to be 
beneficial (Wellein et al., 2009). 

Effective evaluation provides clarity on faculty members and their progress towards 
specific goals and targets. A range of evaluations is needed to gather a comprehensive view of 
faculty instruction (Wellein et al., 2009). According to Wellein et al. (2009), self, specialist, 
student, and peer evaluation formats can be effective. Different formats provide different 
perspectives, thereby offering a holistic view of faculty. Each type of evaluation has unique 
objectives. The purpose of specialist review is to evaluate a specific skill or skill set (Wellein et 
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al., 2009). Through expert supervision, faculty may receive feedback that allows for exploration 
of new teaching material and teaching requirements. Peer evaluations have been found to 
increase faculty implementation of active learning and critical thinking skills within lectures and 
lessons (Wellein et al., 2009). The reflective model used during self-evaluation allows faculty 
members to participate in critical inquiry. Finally, student evaluations have been found to be one 
of the most commonly utilized methods to assess faculty performance (Wellein et al., 2009). 
Student evaluations can provide information regarding teacher effectiveness as well as teacher 
preparedness (Wellein et al., 2009). Effective evaluation processes should be designed to affect 
teaching and resulting learning and to communicate this evidence in a public forum. 

Evaluations of Online Faculty 

While there is an extensive body of research related to the evaluation of faculty in 
traditional settings, there have been fewer studies examining online faculty members’ self- 
reported perceptions of evaluation processes. Due to the rapid growth of online education, the 
existing evaluation scales, such as those used in traditional instructional settings, have been 
questioned (Berk, 2013; Eskey & Schulte, 2012; Hathom & Hathorn, 2010; Mandernach, 
Donnelli, Dailey, & Schulte, 2005; Rothman et al., 2011; Schulte, 2009; Tobin, 2004). As these 
evaluation tools were challenged, concern grew that their relevance and efficiency within the 
online classroom may not be accurate, effective, or sufficient (Berk, 2013). Creasman (2012) 
identified a number of differences in instruction in the online environment. Such differences 
include the asynchronous style of environment, the non-linear forums that allow students to 
participate in several discussions at one time, student-teacher communication, and an increased 
volume of information. Because of this, the complexities of the online environment must be 
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considered when creating and implementing evaluations for online educators (Berk, 2013; 
Harrington & Reasons, 2005; Loveland, 2007). What constitutes an “effective evaluation 
process” for online faculty is an area that continues to require additional study. 


Faculty Engagement in the Evaluation Process 

Although there is a need for more literature on the evaluation processes of online faculty, 
Baran, Correia, and Thompson (2011) claimed that institutions of higher education should 
consider “teachers as adult learners who continuously transform their meaning of structures 
related to online teaching through a continuous process of critical reflection and action” (p. 421). 
This suggests that an effective evaluation of faculty need not only lie in the hands of 
administrators. Indeed, “The process of teacher evaluation in institutions of higher education 
should be an organic whole” (“A preliminary look,” 2005, p. 49). Critical reflections from 
faculty are integral to the development of a comprehensive evaluation program. 

Successful performance evaluation is found in evaluations that are both acceptable to the 
rater and ratee (Szeto, 1994). Furthermore, for faculty members, and ultimately their students, to 
be invested and engaged in their community of practice, they must understand how and why they 
are evaluated. They must feel a degree of commitment to the evaluation process and to 
improving teacher performance. Research has shown that using a bottom-up process will allow 
faculty to obtain a sense of ownership in the standards and evaluation process; thereby, creating 
substantial legitimacy amongst faculty members (Galluscio, 1998). This serves in contrast to a 
top-down model, which relies heavily on administrator evaluation. Commitment to a common 
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community and an increased acceptance of evaluation can be enhanced by involving faculty in 
the development of evaluation procedures (Szeto, 1994). Further research is required to 
determine faculty’s perceptions of the characteristics of an effective online instructor, ideal 
evaluation methodologies, as well as concerns about their evaluation. 


Context 

The online full-time faculty model at the university where the study took place is atypical 
of most institutions with online programs. While many institutions are initiating online full-time 
faculty, location, work requirements, and faculty oversight make this model unique. The model 
includes undergraduate and graduate instructors teaching online in a rolling enrollment program. 
Each instructor teaches approximately four courses at a time and has four computer monitors to 
view documents, assess student work, note phone calls, and engage with students in the 
discussion forums. Notwithstanding their status as online faculty, instructors hold office hours 
eight hours a day Monday through Friday in a building with other online full-time faculty 
members, as well as students’ counselors and support staff. The faculty members are responsible 
for responding to student calls, emails, and messages, as well as grading assignments, guiding 
classroom instruction, and preparing curricular materials for dissemination in the classroom 
during office hours. They are also expected to communicate with traditional faculty and students’ 
counselors. Instructors are encouraged to participate in professional development opportunities, 
as well as scholarly activities, including research and publication. Each faculty member reports 
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to a supervisor and director who conduct informal weekly and quarterly reviews, as well as a 
formal annual review. 

The program has relied substantially on supervisor evaluation of faculty with an analysis 
of at least one course taught per quarter. The quarterly review process was conducted by 
supervisors and served as a convention to formatively assessment and improve practice. The 
document under review in this study included a list of 25 criteria related to the areas of 
participation, engagement, and facilitation; grading and feedback, classroom management; and 
personal development and relationships. Faculty members were rated by supervisors as “met,” 
“partially met,” or “did not meet” for all 25 criteria. The supervisor was expected to offer 
documentation along with the ranking. A summative evaluation of performance was offered at 
the conclusion of the document where the online full-time faculty member was ranked as 
“exceptional,” “good,” or “needs improvement.” 

Because communities of practice suggest the need for thorough, rounded evaluations to 
inform teacher practice and growth, the program described in this study underwent a formal 
program evaluation to explore the evaluation processes of online full-time faculty members. The 
research team established a single objective: to collect feedback from online full-time faculty 
members regarding how they are evaluated and to use this data to improve the university’s online 
full-time faculty quarterly evaluation processes. The research question for the present study 
asked, “To what degree does the university’s online full-time faculty’s evaluation processes align 
with what online instructors perceive as useful and supportive of their efforts as teachers?” 


Participants 
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The research team involved stakeholders directly invested in the development of faculty 
and the impact of teaching on student learning in the online environment. There were six 
members on the research team, including directors, supervisors, and faculty. Three of the 
researchers were part of the administrative team who directly evaluated faculty. The researchers 
were clear in both purpose and expectation of this study, namely that the process was for 
research purposes and improvement initiatives for the online full-time faculty department. 
Researchers exercised transparency by informing participants that the survey was anonymous 
and no one would be able to establish the identities of those involved. 

In the first quarter of 2014, all 169 online full-time faculty members at a large university 
in the Southwest were invited via email to participate in a survey. One hundred and eighteen of 
the 169 faculty participated in the survey. This is a response rate of 69.8%. Notably, the response 
rate may have been influenced by the small-scale pilot study administered prior to the larger 
survey sent to faculty. 

Of the 118 faculty who responded to the survey, 41.53% had been teaching at the 
university level for 2-5 years, 44.07% had been an online full-time faculty member at this 
university for 2-5 years (zero had been in this position at the university for more than five years 
because the position was not created until 2010). The study participants included faculty from 
the doctoral, education, arts and sciences, theology, and business colleges teaching 
undergraduate, masters, and doctoral level courses. The research team members opted not to 
collect additional demographic information on categories like gender, age, racial and ethnic 
identity, or religion because of the researchers’ intimate knowledge of the faculty, ultimately 


ensuring participant anonymity and reducing the potential for researcher bias. 
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METHODS 

A small-scale pilot survey was disseminated via email to a random stratified group of 44 
online full-time faculty members from each college at the university. Survey Monkey, a web- 
based survey service, was used to administer the instrument. The survey was primarily 
qualitative in nature, asking open-ended questions and was distributed prior to the large-scale 
study to identify gaps in the survey instrument. Results from the pilot study resulted in two 
changes: the research team clarified the wording on one of the questions and included the current 
document used by supervisors to evaluate faculty for reference. 

A revised follow-up survey was sent via email and Survey Monkey to all 169 online full¬ 
time faculty members at the university where the study took place. Faculty members were told 
that their participation was voluntary and anonymous. Further, faculty members were not 
required to answer every question on the survey. Participants completed the survey in 
approximately 20 minutes and were given two weeks to complete the survey until the link was 
closed. 

The survey asked descriptive questions regarding online teaching and the evaluation 
processes of online instructors. The instrument was divided into three sections, including: (1) 
perceptions of the roles of online faculty, (2) perceptions of teaching evaluations, and (3) 
perceptions of the current evaluation processes for online full-time faculty. The first section, 
perceptions of the roles of online faculty, contained items related to qualities of an effective 
online instructor, identity as an online instructor, and areas of growth or opportunity as an online 
instructor. The second section, perceptions of teaching evaluations, included items related to the 
importance of evaluating online instructors, most beneficial types of evaluations, and an ideal 
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evaluation of teaching. The third section, perceptions of the current evaluation processes for 
online full-time faculty, contained items regarding the current evaluation processes for faculty at 
this university, elements to revise in the current process, and the effectiveness of the current 
process. There were 11 descriptive questions in total. 

The descriptive survey questions were analyzed qualitatively through the content analysis 
method (Carney, 1972; Holsti, 1968, 1969; Krippendorff & Bock, 2008). The analysis process 
was systematic and purposeful. The team reviewed 11 descriptive survey responses from 118 
full-time online faculty members, highlighting and notating each unit of analysis relevant to the 
research question. Units of analysis included descriptive words, phrases, and sentences. After the 
initial analysis, similar units were combined. These units were then collapsed into other larger 
categories based on similar content or redundancies. After this step, key words or phrases from 
the units were extracted, resulting in a set of code or categories for each descriptive question. 

The process continued until all relevant units were grouped or regrouped with similar units and 
labeled with a code (Krippendorff & Bock, 2008). The team then identified robust themes by 
counting instances for frequency (Krippendorff & Bock, 2008). For the purposes of this analysis, 
any code with more than seven units was considered robust. There was a decline after seven 
units for each code; thus, the selection of seven was intentional, not arbitrary. Codes with less 
than seven units were considered weak and not included in the findings and analysis. 

The team analyzed each set of survey responses independently to develop codes with as 
little bias as possible, focusing on the words scribed by survey respondents. The six researchers 
each shared his or her codes through a coding workshop designed to ensure intercoder reliability 


(Miles & Huberman, 1994; Neuendorf, 2002). The workshop afforded researchers the 
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opportunity to identify points of conflict or communion in the coding process, to move codes 
into new categories, to alter the language of categories if needed, and to agree upon robust codes. 
For example, one of the researchers had identified a code in her individual coding session; 
however, after group discussions, it was determined that the label was not specific enough. As 
such, the group developed a new label in concert to describe the phenomenon. A series of robust 
codes emerged from the coding workshop and are explicated in the findings and analysis section. 
Findings and Analysis 

In this paper, robust codes from three of the 11 descriptive questions are detailed. For 
detailed findings from one specific survey question, please reference Author 1 et al. (in press). 
These three questions were selected for discussion because they most align with the study’s 
primary research question: “To what degree does one university’s online full-time faculty’s 
evaluation processes align with what online instructors perceive as useful and supportive of their 
efforts as teachers?” The three survey questions analyzed included: (1) “Do you feel the current 
supervisor evaluation document is useful to you? Why or why not?”; (2) “Do you feel the current 
supervisor evaluation process supports your work as a teacher? Why or why not?”; and (3) “If 
you could envision the ideal process to evaluate your teaching, what might that process look 
like? How frequently would you be evaluated?” 

For the survey question, “Do you feel the current supervisor evaluation document is 
useful to you? Why or why not?,” 57 participants responded with positively aligned comments, 
40 with negatively aligned comments, 14 stated there were positive and negative elements to the 
document, and eight stated they had never seen the document. No robust codes emerged from 
responses with both positive and negative comments regarding the document. See Table 1. 
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The most robust code for positively aligned responses was that the document identifies 
areas to improve and reflect. There were 23 units in this code. When describing whether the 
document was useful, faculty expressed comments such as, “Gives me areas to improve,” “sets 
goals and reflects on my progress,” “it points out areas of improvement,” “reminds me of 
strengths and weaknesses,” “helps find area of improvement,” and “indicates where 
improvement can be made.” Faculty members’ comments suggested that the majority of online 
full-time faculty at this university believed the current supervisor evaluation document offered 
opportunities to reflect on specific areas of their teaching that needed improvement. 

The second most robust code for positively aligned comments was that the document 
established clear expectations for their performance. There were 15 units in this code. When 
describing whether the document was useful, faculty expressed comments such as, “Useful to 
meeting all my expectations,” “allows me to refer back to my goals and expectations regularly,” 
“useful reflection on what is required by the university,” and “good to know how the university 
feels that one is doing.” Faculty members’ comments established that the document provided 
clarity into expectations as it relates to their job performance and the requirements of the 
university’s online full-time faculty department. 

The most robust code for negatively aligned responses was that the document focused on 
operational tasks, not teaching quality. There were 12 units in this code. When describing why 
the document was not useful, faculty expressed comments such as, “It just measures the 
performance of operational tasks,” “it is just a fixation on details,” “does not focus on what is 
really important,” “tremendous amount of emphasis on what you have or have not completed in 
relation to your requirements,” “all it measures is quantifiable data,” “spends too much time on 
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numerics,” “I don’t like the more robotic expectations lists,” and “I already know whether or not 
I have completed administrative tasks.” Faculty members’ comments suggest that some found 
the document helpful for clarifying expectations while others found the document to be heavily 
focused on quantitative, detailed, or “robotic” measurements, as opposed to a holistic portrait of 
the classroom. Further, the data demonstrated that the evaluation document did not emphasize 
teaching quality and “what is really important,” although those faculty did not specific what 
qualified as “important” in the online classroom. 

The second most robust code for negatively aligned responses was that the document is 
top-down or operationally oriented. There were nine units in this code. When describing why the 
document was not useful, faculty expressed comments such as, “Possible manipulation of the 
document to be for or against an employee,” “they are set up for failure,” “instructors feel over 
watched,” and “only tells me what my supervisor saw; no other information is new news.” 
Faculty members’ comments suggested that these nine faculty not only found the document 
misguided and focused on administrators, but also reported an approach that was misaligned with 
the freedoms they associate with membership in the academy, as noted by comments related to 
being “watched” by supervisors. 

Table 1 


Survey Question 

Robust Code 

Number of 
Units 

Do you feel the current supervisor 
evaluation document is useful to 
you? Why or why not? 

Identifies areas to improve and reflect 

23 

Established clear expectations for their 
performance 

15 

Focused on operational tasks, not 
teaching quality 

12 

Top-down or operationally oriented 

9 
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For the following question, “Do you feel the current supervisor evaluation process 
supports your work as a teacher? Why or why not?,” 41 answered with positively aligned 
comments, 34 with negatively aligned comments, and 23 with comments stating both positive 
and negative elements of the process. See Table 2. 

The most robust code with positively aligned responses was identifies areas for 
improvement or opportunity in instruction. There were 21 units in this code. When describing 
why the current process supported their teaching, faculty expressed comments such as, “Helps 
improve my teaching and practices,” “directions on areas that I can become better in,” “makes 
me a better teacher,” “challenges me to be better,” “lets me know what I’m doing well,” “shows 
me what I can improve on,” “provides ideas for instructional improvement,” and “gives 
instructional feedback.” Faculty members’ comments suggested that the current process 
uncovered specific areas or “directions” an instructor can move to improve their instructional 
techniques and generally “be better.” The focus on being challenged to move forward in their 
online teaching was strong. The faculty stressed how the process helped them as individual 
instructors improve, perhaps ultimately influencing student learning. 

The second most robust code with positively aligned responses was identifies degree to 
which expectations are met. There were seven units in this code. When describing why the 
current process supported their teaching, faculty members expressed comments such as, “Ensure 
I’ve met necessary components,” “meeting my expectations,” “meeting my grading 
expectations,” and “reviews time spent in cloud, met requirement.” Similar to the first research 


question regarding the effectiveness of the evaluation document, faculty noted that the process 
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illuminated whether the expectations of their work as online full-time faculty members were met 
and the degree to which they were met. 

The most robust code with negatively aligned responses was focuses on expectations 
rather than instructional practices. There were 13 units in this code. When describing why the 
current process did not support their teaching, faculty expressed comments such as, “A faculty 
checklist,” “a task scorecard,” “review of the standard expectations,” “’checklisty,”’ “no 
feedback on actual teaching practices,” “no focus on pedagogical content knowledge of 
instructor,” and “not actually evaluating the teaching or potential teaching of an instructor.” 
These faculty members expressed frustration or anxiety related to the lack of emphasis placed on 
content knowledge and the strong emphasis on the “checklist” structure of the evaluation 
document. 

The other equally robust code with negatively aligned responses was the role of 
management and policies in process. There were 13 units in this code. When describing why the 
current process did not support their teaching, faculty expressed comments such as, 
“Management documentation of my performance,” “I never see the results,” “those that review 
me do not teach my course,” “not connected to learning experience, only to administration,” and 
“a piece of paper is just a piece of paper.” The faculty members who were concerned with the 
process noted concerns with the procedures associated with the dissemination of the document, 
as well as the experience of the supervisor completing the evaluation. 

Several responses offered both negatively and positively aligned responses, creating one 
robust code. The robust code in this category was that evaluation processes need to be revised. 


There were 18 units in this code. When describing why the current process was somewhat 
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supportive of their teaching, faculty members expressed comments such as, “One should be able 
to achieve excellence in reviews,” “better served with specific reviews based on subject being 
taught,” “too micro-focused,” “helpful if done in a way to help us grow as educators,” “should 
include ways to measure new and innovative teaching strategies,” and “training would be more 
effective.” These faculty members reported an appreciation for an element of the process but 
explained that it was inadequate because there were opportunities for revision in the following: 
an improved scoring system, more content-focused comments from evaluators, and enhanced 
procedures used to disseminate the evaluation. One individual offered a different approach, 
specifically a faculty training system, rather than supervisor evaluation. 

Table 2 


Survey Question 

Robust Code 

Number of 
Units 

Do you feel the current supervisor 
evaluation process supports your 
work as a teacher? Why or why not? 

Identifies areas for improvement or 
opportunity in instruction 

21 

Evaluation processes need to be 
revised 

18 

Focuses on expectations rather than 
instructional practices 

13 

The role of management and policies 
in process 

13 

Identifies degree to which 
expectations are met 

7 


For the following question, “If you could envision the ideal process to evaluate your 
teaching, what might that process look like? How frequently would you be evaluated?”, the most 
robust code was that evaluations should focus on growth of the instructor and students. There 
were 33 units in this code. When describing the ideal process, faculty expressed comments such 
as, “Desire for growth,” “less task-y or checklist-y,” “more qualitative and personal,” “given 
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specifics on how to improve,” “evaluate of use of higher order thinking,” “focus on growth of 
employee,” “promote ongoing growth,” “show areas of growth,” “qualitative rather than 
quantitative,” and “challenge critical thinking and deeper thinking.” Faculty reported that the 
ideal process should be qualitative, personal, less focused on a checklist, and intended to grow 
the faculty’s online teaching techniques and ensure student learning. Those who envisioned a 
new system expressed their “ideal” process in contrast to the system currently in place to 
evaluate their teaching and expressed a desire for qualitative, holistic, and inquiry-based 
feedback. 

The second most robust code was that administrators should select evaluators that can 
effectively evaluate courses. There were 14 units in this code. When describing the ideal process, 
faculty expressed comments such as, “Supervisors may not have the training or experience in my 
specific field to provide adequate assessment,” “evaluated by a subject matter expert,” “faculty 
to meet with one another to share best practices,” “evaluators who know the content to evaluate a 
class,” and “someone who is capable of instructing my content should evaluate me.” These 
faculty members noted interest in being reviewed by a peer or supervisor with subject matter 
expertise and the ability to share best practices within a particular content. 

Of those who noted frequency in their responses, 26 or 41% preferred bi-annual 
evaluations. Twenty or 32% preferred annual supervisor evaluations. Seventeen or 27% 
preferred quarterly supervisor evaluations. 

Table 3 


Survey Question 

Robust Code 

Number of 
Units 

If you could envision the ideal 
process to evaluate your teaching. 

Focus on growth of the instructor and 
students 

33 
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what might that process look like? 
How frequently would you be 

Select evaluators that can effectively 


evaluated?” 

evaluate course 

14 


DISCUSSION 

Findings from this study have several implications. Above all, online full-time faculty 
members want to grow as instructors. Although this is not surprising data, it has four functions. 
First, it provides an initial baseline for understanding how online full-time faculty members 
perceive evaluation processes. Online full-time faculty departments are relatively new 
phenomena in higher education, specifically the model used at the university described in this 
study. Because this department is unique in regards to proximity and oversight, the baseline 
serves as the foundation for universities that resemble the one discussed in this study who are 
developing evaluation processes for online faculty. 

Second, online full-time faculty members in this study rarely expressly focused on 
modality. “Growth” was stated in general terms regarding a desire to be a better pedagogue for 
student learning, not necessarily a desire to be a better instructor in the online modality. This 
does not mean that the faculty conceived of their roles or evaluation as online instructors as 
unimportant. This does mean, however, that this population preferred evaluations focused on 
content and teaching practices. The study’s contributes the notion that while differences in online 
and traditional teaching may be clear to some stakeholders, it may not be tangible or significant 
to online full-time faculty. In this setting, modality did not appear to be a factor in faculty’s 
perceptions of evaluation. Faculty members strongly emphasized general instructional 
improvements in their responses. Online full-time faculty’s perceptions, specifically the robust 
codes, are arguably applicable to all instructors, regardless of modality. Stakeholders can use this 
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knowledge when developing criteria for online faculty evaluations. More research is needed to 
understand what it means to encourage growth in online instructors and what role modality plays 
in instructors’ concepts of online teaching, particularly in settings like the one described in this 
study. 

Third, the emphasis on growth suggests that online full-time faculty members are eager to 
refine their skills as instructors and want the evaluation tool to mirror their desire for growth, not 
only capture the degree to which they have or have not met job expectations. Respondents 
emphasized a desire to be challenged and to improve their teaching skill set, while 
deemphasizing the role of checklists, tasks, and expectations. Supervisors in computer-mediated 
teaching departments can use this information when disseminating evaluation results and when 
developing plans to lead their teams towards instructional improvements. 

Finally, while faculty members in this study identified positive elements of the process, 
some felt that the current evaluation document was not sufficient, the current evaluation process 
did not fully support their work as faculty, and the ideal process for evaluating online full-time 
faculty did not align with the process currently in place. Faculty expressed concerns that the 
current document and evaluation process weighed heavily on expectations or quantitative data, 
as well as noted the limitations of the top-down approach of supervisor evaluation and evaluation 
by individuals without specific content knowledge. This finding has implications for 
administrators in computer-mediated teaching departments when developing evaluations, ideally 
focused on qualitative or holistic measures disseminated through a rounded approach by content 


experts. 
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Although the negatively aligned comments may appear to be simply critiques of the 
department’s evaluation processes, the research team sees otherwise. The department was 
established in the latter half of 2010, and it is an organism that seeks improvement in all 
programmatic areas from curriculum to assessment of students to faculty working conditions to 
evaluation processes. This survey data confirms the need to continue to improve the 
department’s evaluation document and process with an emphasis on the growth of faculty. 
Because of the department’s distinct structure, there is a built-in community of practice, one that 
focuses on reflection and innovation (Lave & Wenger, 1991; Wenger, 2000). It is incumbent 
upon the department and other computer-mediated teaching departments to ensure that 
evaluation documents and processes align with the community and to understand how physical 
proximity might influence perceptions of online teaching evaluations. 

These findings led the research team to reflect upon next steps, ultimately choosing to 
collect additional data from faculty. The research team deemed it necessary, based on the 
outcome of the present study, to establish a focus group and workshop to further understand the 
strengths and weaknesses of the current evaluation system. This process was designed to 
integrate researchers, faculty, and administrators, working collaboratively to revise the 
evaluation document and process. These next steps are multi-fold and rooted in the framework 
and findings of this study. The focus group and workshop were intended to enhance faculty’s 
community of practice in order to lead to enhanced knowledge of online teaching, greater 
collaboration, and improved student learning (Garrison, Anderson, & Archer, 2000, 2010; Lave 
& Wenger, 1991; Wenger, 1998, 2000). Additionally, as the data demonstrated, a significant 


number of faculty members expressed concern about the emphasis placed on quantitative 
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measures, as well as the individual completing the evaluation. The research team identified the 
opportunity for growth in the evaluation processes of online full-time faculty and chose to gather 
additional faculty input on these concerns. This new study aligns with the literature suggesting 
that faculty evaluations should be comprehensive and involve faculty insight, as well as the 
literature stating that faculty inquiry and reflection can lead to greatest growth in student learning 
(MacMillan et al., 2010; Wellein et al., 2009). 

Because of the study described in this manuscript, as well as the focus group and 
workshop study, the online full-time faculty department intends to revise the evaluation 
instrument and process to align with faculty’s perceptions and to meet their needs as scholars of 
teaching and learning (Boyer, 1990; Glassick et al., 1997). The newly developed evaluation 
instrument will seek to include useful components from the original document, while 
incorporating components focused on teacher effectiveness, growth, and the opportunity for 
evaluators to provide qualitative, holistic feedback. In addition, the new evaluation document 
and process will use peer, supervisor, and self-evaluations rather than focus solely on supervisor 
evaluation. As the new document and process are developed, implemented, and refined, 
additional research on online full-time faculty’s perceptions of evaluations will follow. 



“A DESIRE FOR 

GROWTH” 98 


References 

A preliminary look at systems of evaluating teachers in institutions of higher education and at 
evaluation methods. (2005). Chinese Education & Society, 38(5), 48-60. 

Allen, I. E., & Seaman, J. (2013). Changing course: Ten years of tracking online education in the 
United States. Retrieved from 

http://www.onlinelearningsurvey.com/reports/changingcourse.pdf 
Arreola, R. A. (1979). Strategy for developing a comprehensive faculty evaluation system. 
Engineering Education, 12, 239-244. 

Arreola, R. A. (1986). Evaluating the dimensions of teaching. Instructional Evaluation, 8(2), 4- 
14. 

Arreola, R. A. (1995). Developing a comprehensive faculty evaluation system. Bolton, MA: 
Anker Publishing Company. 

Arreola, R. A. (2000a). Developing a comprehensive faculty evaluation (2 nd ed.). Bolton, MA: 
Anker Publishing Company. 

Arreola, R. A. (2000b). Interview. The Department Chair, 11(2), 4-5. 

Arreola, R. A., Aleamoni, L. A., & Theall, M. (2001). College teaching as meta-profession: 
Reconceptualizing the scholarship of teaching and learning. Paper presented at the 9 th 
Annual American AAHE Conference on Faculty Roles and Rewards, Tampa, FL. 

Baran, E., Correia, A., & Thompson, A. (2011). Transforming online teaching practice: Critical 
analysis of the literature on the roles and competencies of online teachers. Distance 
Education, 32(3), 421-439. doi:10.1080/01587919.2011.610293 


Berge, Z., & Meyers, B. (2000). Evaluating computer mediated communication courses in higher 



“A DESIRE FOR 

GROWTH” 99 


education. Journal of Educational Computing Research, 23(4), 431-450. 

Berk, R. A. (2013). Face-to-face versus online course evaluations: A “consumer's guide” to 
seven strategies. Journal of Online Teaching and Learning, 9(1), 140-148. 

Boyer, E. L. (1990). Scholarship reconsidered: Priorities of the professoriate. Princeton, NJ: 

Carnegie Foundation for the Advancement of Teaching. 

Carney, T. F. (1972). Content analysis. Winnipeg: University of Manitoba Press. 

Clark, R. E. (1989). Evaluating distance learning technology. Paper presented at United States 
Congress, Office of Technology Assessment, Washington DC. 

Creasman, P. A. (2012). Considerations in online course design. Manhattan, KS: The IDEA 
Center. Retrieved from 

http://www.theideacenter.org/sites/default/files/idea_paper_52.pdf 
Eskey, M. T., & Schulte, M. (2012). Comparing attitudes of online instructors and online 

college students: Quantitative results for training, evaluation and administration. Online 
Journal of Distance Learning Administration, 15(5). 

Galluscio, E. H. (1998). Performance evaluation in an academic department. The Psychologist- 
Manager Journal, 2(2), 43-48. doi:10.1037/h0095847 
Garrison, D. R., Anderson, T. & Archer, W. (2000). Critical inquiry in a text-based environment: 
Computer conferencing in higher education. Internet and Higher Education, 2(2-3), 87- 
105. 

Garrison, D. R., Anderson, T., & Archer, W. (2010). The first decade of the community of 


inquiry framework: A retrospective. Internet and Higher Education, 13, 5-9. 



“A DESIRE FOR 

GROWTH” 100 

Glassick, C. E., Huber, M. T., & Maeroff, G. I. (1997). Scholarship assessed: Evaluation of the 
professoriate. San Francisco: Jossey-Bass. 

Hathom, L., & Hathorn, J. (2010). Evaluation of online course websites: Is teaching online a tug- 
of-war? Journal of Educational Computing Research, 42(2), 197-217. 
doi:10.2190/EC.42.2.d 

Holsti, O. R. (1968). Content analysis. In G. Lindzey & E. Aaronson (Eds.), The handbook of 
social psychology. Reading, MA: Addison-Wesley. 

Holsti, O. R. (1969). Content analysis for the social sciences and humanities. Reading, MA: 
Addison-Wesley. 

Krippendorff, K. H., & Bock, M. A. (2008). The content analysis reader. Thousand Oaks, CA: 
Sage. 

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. 
Cambridge, UK: Cambridge University Press. 

MacMillan, M., Mitchell, M., & Manarin, K. (2010, March 1). Evaluating teaching as the first 
step to SoTL. Paper presented at SoTL Commons Conference, Statesboro, GA. 
Mandernach, J. B., Donnelli, E., Dailey, A., & Schulte, M. (2005). A faculty evaluation model 
for online instructors: Mentoring and evaluation in the online classroom. Online Journal 
of Distance Learning Administration, 8(3). 

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis. Thousand Oaks, CA: Sage. 
Moore, M., & Kearsley, G. (2012). Distance education: A systems view of online learning. 
Belmont, CA: Wadsworth CENGAGE. 

Neuendorf, K. A. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage. 



“A DESIRE FOR 

GROWTH” 101 


practices from f2f to online environments. Electronic Journal of E-Learning, 8(1), 1-12. 

Ragan, L. C. (2009). 10 principles of effective online teaching: Best practices in distance 
education. In Faculty Focus. Retrieved from http://www.facultyfocus.com/free- 
reports/principles-of-effective-online-teaching-best-practices-in-distance-education/. 

Rothman, T., Romeo, L., Brennan, M., & Mitchell, D. (2011). Criteria for assessing student 
satisfaction with online courses. International Journal for e-Learning Security, 7(1-2), 
27-32. 

Schulte, M. (2009). Efficient evaluation of online course facilitation: The “quick check” policy 
measure. The Journal of Continuing Higher Education, 57, 110-116. 

Smith, M. K. (2003). Communities of practice. The Encyclopedia of Informal Education. 
Retrieved from www.infed.org/biblio/communities_of_practice.htm. 

Smith, M. K. (2009). Jean Lave, Etienne Wenger, and communities of practice. The 

Encyclopedia of Informal Education. Retrieved from http://infed.org/mobi/jean-lave- 
etienne-wenger-and-communities-of-practice/. 

Szeto, W. F. (1994). Performance evaluation among faculty across academic disciplines at a 
selected university. Paper presented at the perceptions of Annual Meeting of the Mid- 
South Educational Research Association Faculty, Nashville, TN. 

Tobin, T. J. (2004). Best practices for administrative evaluation of online faculty. Online Journal 
of Distance Learning Administration, 7(2). 

Wellein, M. G., Ragucci, K. R., & Lapointe, M. (2009). A peer review process for classroom 


teaching. American Journal of Pharmaceutical Education, 73(5), 1-7. 



“A DESIRE FOR 

GROWTH” 102 


Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge: 
Cambridge University Press. 

Wenger, E. (2000). Communities of practice and social learning systems. Organization, 7(2), 
225-246. 

Wenger, E. (n.d.). Communities of practice: A brief introduction. Retrieved from http://wenger- 
trayner.com/theory/ 



